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ABSTRACT 

This  paper  is  concerned  with  gradient  estimation 
techniques  for  steady>8tate  perfwmance  measures  a»> 
sodated  with  regenerative  stochastic  processes.  The 
principal  enq>hasis  is  on  the  discussion  of  conditions 
under  which  likelihood  ratio  methods  and  infinitesi* 
mal  perturbation  analysis  techniques  are  valid. 

1  INTRODUCTION 

This  paper  is  concerned  with  using  simulation  to  esti> 
mate  gradients  of  steady-state  performance  measures 
in  the  regenerative  process  setting.  This  turns  out  to 
be  quite  a  rich  class  of  stochastic  processes  ficom  an 
applications  viewpoint,  encompassing  all  irreducible 
recurrent  discrete  state  space  Markov  processes  as 
well  as  a  variety  of  more  general  discrete-event  sys¬ 
tems.  In  particular,  a  large  class  of  generalised  semi- 
Markov  processes  may  be  made  regenerative  by  using 
'‘splitting”  ideas  from  the  theory  of  Harris  recurrent 
markov  chains;  see,  for  example,  Glynn  (1989). 

Two  different  gradient  eatiinatiaa  algorithms  are 
explored  in  this  paper;  Ukelibood  ratio  gradient  esti- 
mationa  and  estimationa  baaed  on  hafiiiitesifnal  per¬ 
turbation  analysis  (IPA).  The  focos,  in  this  paper,  is 
on  the  tools  needed  to  rigMOusly  verify  the  mathe¬ 
matical  validity  of  these  techniques  in  the  regenera¬ 
tive  setting. 

2  STEADY-STATE  LIKELIHOOD 
RATIO  GRADIENT  ESTlM^mON 

Let  IV  =  (IV„  :  n  >  0)  be  a  reaLvalned  stochas¬ 
tic  sequence.  For  each  f  €  A  =  (a,k),  let  be  a 
probability  distribution  on  the  path  space  of  W.  We 
assume  that  there  exists  a  non-decreasing  sequence 
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T  =  r(n)  :  n  >  0)  of  random  times  such  that 
r(0)  =  0  and; 

(Al)  Under  distribution  P«,  IV  is  a  non-  delayed  re¬ 
generative  sequence  with  respect  to  T,  for  each  ^  €  A. 

Ut  y.  =  =  ESfcL.iliyjl,  ad 

Tn  =  T{n)  —  T{n  -  1).  If  £${•)  is  tne  expectation 
operator  associated  with  P*,  we  require  that  E$Y^i  < 
00  and  £«r\  <  oo,  in  which  case 

(2.1) 

>s0 

as  n  -*  oo,  where  a(d)su(d)/f(d)  and  u(0)  = 

£,y„f(d)  = 

Relation  (2.1)  implies  that  a($)  may  be  inter¬ 
preted  as  the  steady-state  mean  of  IV  under  P#.  Our 
goal  is  to  develop  an  estimation  methodology  for 
ar'(fo)  for  (fixed)  do  €  A.  Since  a  s  u/f,  it  is  clearly 
sufficient  to  devebp  estimators  for  u'(do),P(do), 
u(lo),  and  /(do)-  Of  course,  ti(do)  and  f(do)  can  eas¬ 
ily  be  estimated  via  sample  means  formed  from  i.i.d. 
copies  of  Yi  and  ri  generated  under  P«,.  The  greater 
challenge  is  to  develop  estimators  for  ii'(do)  and  P(do). 

To  accomplish  this  task,  we  inumr  that  there 
exists  a  e-field  G  for  which  Yi  and  are  {l-meaaurable 
and  such  that  P«  is  absolutely  continuous  with  respect 
to  P«,  on  G.  Consequently,  for  each  d  €  A,  we  may 
represent  «(d)  and  f(d)  aa 

u(d)=£YiI(d) 

f(d)  =  EnX(#). 

where  E(-)sE«,(-)  and  L(d)  is  the  likelihood  ratio 
(Radon-Nikodym  derivative)  of  P$  with  respect  to 
P«,.  Suppose  that  X(-)  is  P«,  an.  differentiable  at 
do.  If 


(A2)  (y(I(ffo  +  M  -  i(«o)/A  :h>0,8o  +  he  A) 
and  (r(L(flo  +  L)-  L{8o))/h  ;  A  >  0,  So  +  *  €  A)  are 
uniformly  integrable  under 

then  it  follows  that 

u'(So)  =  EYiL'{0o) 

f'(So)  =  EnL\0o). 

Hence,  under  (A2),  u'(So)  and  f'(So)  can  be  c<suis- 
tently  estimated  via  sample  means  formed  from  i.i.d. 
copies  of  yi£'(So)  and  tiL'(0o)  generated  under 
This  then  solves  the  problem  of  estimating  a'(So)  con¬ 
sistently. 

The  key,  from  a  mathematical  viewpoint,  is 
therefore  to  verify  (A2). 

2.1  Harris  Chains 

A  large  class  of  regenerative  systems  are  derived  from 
discrete-time  Markov  chains  2  =  {2n  ■  n>  0)  living 
on  some  general  state  space  5.  Assume  that  there 
exists  a  subset  A  C  5,  A  >  0,  a  positive  integer  m  >  1, 
and  a  probability  distribution  <fi  such  that: 

(A3)  i)  Ptl2n  €  A  infinitely  often  \2q  =  z]  =  1  for 
fl€A,z€5, 

ii)  Pi[2m  €  dz|Zo  =  >  A^dz)  for  z  €  A, z  € 

5,d6  A. 

Condition  (A3)  guarantees  that  for  Z  is  a  Harris  re¬ 
current  Markov  chain  under  P«  for  each  0  €  A.  It 
is  well  known  that  (A3)  can  be  expbited  to  obtain 
regenerative  structure  fx  2.  Suppose  that  2o  has 
distribution  Now,  observe  that  the  minoruation 
condition  (A3)  ii)  guarantees  that  for  z  €  A  we  can 
write 

P,l2„,€dil2o  =  x] 

=  A^dz)-|.(l-A)g(l,z,dx) 

where  Q(8,x,  •)  it  x  probability  distribution  on  5  for 
each  8  €  A  and  z  €  A.  Hence,  each  time  2  visits 
A,  there  is  a  probability  A  >  0  that  the  chain  will  be 
distributed  according  to  ^  m  time  units  later,  thereby 
constituting  a  legeneratioii.  More  precisely,  we  can 
introduce  a  seqiuenee  (%  :  n  >  0)  of  "coin  flip”  r.v.’s 
such  that  fM  -  0  iHienevet  Zn  ^  A,  and  s  1(0) 
whenever  Zn  €  A  and  a  ”succeasflii*  (*nnsucceasful”) 
coin  flip  occurs.  Then,  the  regeneratioo  time  can 
be  defined  as  r|  =  tn/{n  >  m  :  ~  !}• 

For  a  given  function  /  :  5  -»  fZ,  let  IF),  = 

Then,  it  is  evident  that  if  JF«ri  <  oo  and  EfYi  <  oo, 

1 

ai§)tE,Yi/E,n  P,  a.s.  (2.3) 

”  >s0 


as  n  — *  OO.  To  calculate  the  derivative  o'(ffo)i  we  have 
already  shown  that  (A2)  b  the  key. 

To  obtain  a  likelihood  ratio  estimator,  we  will 
assume  that  there  exist  “densities”  p(0,  z,  y),  q{0,  z,  y) 
such  that: 

(A4)  i)  For  z  ^  A,y  €  S,fl  €  A,P(ff,z,dy)  = 
p{8,x,y)P{8o,*,dy) 

ii)  For  z  €  A,y  €  S,0  €  A,Q{0,z,dy)  - 
<l{0,*,y)Q(0o,x,dy) 

Assumptions  (A3)  ii)  and  (A4)  together  guar¬ 
antee  that  there  exist  densities  pn(0,  x,  y)  such  that 
P#I^n  €  dzjZo  =  z]  s  Pn{0iX,z)Pt,[2n  €  dz\2o  €  *] 
for  €  A,z,z€S. 

To  obtain  a  likelihood  ratio  for  the  path  of  the 
sequence  Xn  =  {2n,T)n)  up  to  tinoe  r^,  we  decompose 
the  path  according  to  appropriately  spaced  visits  to 
the  set  A.  Specifically,  let  5o  =  -m  and  set  5t  = 
in/{n  >  Sk-i  +  m  :  Zn  €  A},  and  let  0  —  inf[k  > 
1  :  T)s^  =  1}.  Then,  the  likelihood  ratio  L{0)  on  the 
«r-field  S  =  <r(2n  :  0  <  n  <  r^)  of  P#  with  respect  to 
P«,  can  be  written  in  the  form 


0 


kBl 


where 


?(^i^5>i^5fffn) 
Pm(®i^S*  I  ^5t4.w) 


tx  k<  0  and 


hi») 


1 

Pwt(^i  Es0 1  Zs0^m') 


Assume  that  the  functions  Ph(-.  x,  y)  and  9(-,  z,  y) 
are  continuously  differentiable  on  A  for  n  >  1  and 
x,y  e  S.  To  verify  (A2),  note  that  Y(L(0o  -h  h)  - 
L(0o))/fk  =  YL'(0  for  some  (  €  [9o.  0o  +  A).  But 


l'(0) 

Cr,-1  0 

£l/(0,Zj,2j^.O  +  ^8i(0) 

isO 


where 


Pm(0,Zs^,2s^+m) 


fox  k<  0  and 


i  '$)  = 


Pm(9,Zs0,Zst+m) 


We  have  used  here  the  fact  that  Pn(,0,x,y)  > 
0(«(^,x,y)  >  0)  whenever  ^  W^.x.y) 

0).  For  c  >  0,  set 

p(z,y)=  sup  b(tf,z,y)| 

^(*,y)=  sup  \p'(e,z,y)\ 

ik  -  sup  \ik(8)\ 

sup  14(5)1, 

I*-*.!!* 

and  observe  that  by  continuity  p{x,  y)  and  4  wiU  typ¬ 
ically  be  close  to  1.  Then, 

■  sup  1(5). 

To  establish  (A2)  requires  proving  that  the  dominat¬ 
ing  r.v.  just  defined  is  intepable.  But  its  expectation 
is  bounded  by 

II  «ip  L(5)l|pi  •  llnll,,  •  II  ^  l/(^k)lllr, 

#-#,!<«  0<*<r, 

•Olnllsa  „^,J#'(^k.^k+l)lll,. 

+  ll^lls.  •llo^^lMIW 

when  Pi, pj,..., PS  >  0  and  p{’‘  +  pj’^+  -  -Hp^'  =  1. 
But 


(SH‘ 


the  other  maxiirmn  terms  can  be  similacly  bounded 
by  sums.  These  expectations,  as  well  as  ||ri||,,  can 
be  bounded  by  standard  Lyapunov  function  methods; 
see  Chapter  15  of  Meyn  and  Tweedie  (1992).  (Note 
that  0  is  geometric  so  ||^|^  is  trivially  finite.) 

The  greater  difficulty  is  posed  by  the  term 
ll»«P|»-#,|<ci(^)l|r- 

ai(r)  =  £:(  sup  Ii(5)’’l^  >  l,Zo  =  x] 

aj(r)  =  r(  sup  Ii(5)'’|^  =  l.Zo  =  x). 
(«-«•!<* 


By  successively  conditioning  and  using  the  strong 
Markov  property  at  the  times  So,  Si , ....  S4_ i ,  it  fol¬ 
lows  that 

II  sup  i(«)||r 

l*-«al<* 

4-1 

i=0 

Hence,  establishmg  (A2)  requires  getting  a  handle  on 
the  functions  qi  and  93.  In  many  applications,  the  set 
A  is  compact .  Continuity  arguments  then  permit  one 
to  control  p-‘ (5,  Zs^ ,Zs,+m)  and  ?(5,  Zs^ ,  Zs»  +m )/ 
Pmi9,Zs^,Zs^+m)  are  bounded.  Furthermore, 

m-1 

E  ]lp(Zi,Zi+iY\Z^  =  z 
,i*o 

is  typically  bounded  over  x  €  A.  Consequently,  the 
key  to  bounding  91  and  93  is  to  get  a  handle  on 

rs,-i 

A(x)4£:  JIp(Z,-,W|Zo=:x 
.>*0 

for  X  €  A*.  We  complete  this  discussion  of  the  verifi¬ 
cation  of  ( A2)  by  providing  a  L]rapunov  function  crite¬ 
rion  for  bounding  h.  Let  K(x, dz)  =  p{x,  zyP0,[Zi  € 
dxjZo  s  x]  and  set  f(x)  s  iir(x,A).  A  standard  ar¬ 
gument  establishes  that  if  we  can  find  a  non-negative 
function  r  satisfying 

f  K(x,dz)f{z)  <  f(x)  -  ty(x) 

Ja» 

for  X  €  A*  and  c  >  0,  the  bound  A(x)  <  r(x)/x  for 
X  €  A*  follows. 

2.2  Stochastic  Recursioiu 

A  large  class  of  Markov  chains  Z  satisfy  stochastic 
recursions  of  the  form 

Zn^l  =  d(Z,.Cf,) 

where  d  :  5  x  S'  -»  5  and  U  =  (Un  :  n  >  0)  is 
a  sequence  of  S'-valued  r.v.’s  that  is  Li.d.  under 
for  each  5  €  A.  In  order  to  guarantee  that  Z  be 
regenerative,  we  require  the  existence  of  a  subset  B 
and  a  family  ip{$,  ■)  of  probability  distributions  such 
that: 


(AS)  i)  P«[Z«  €  B  infinitdy  often  |Zo  =  x]  =  1  for 
x€S,5€  A, 


ii)  Pt\Z\  6  dzjZo  =  r]  =  dz)  for  r  g  5,z  6 
S. 

Clevly,  Ti  =:  mf{n  >  1 ;  Zn-i  €  ^}  is  a  regeneration 
time  for  Z.  (In  fact,  Z  satisfies  (A3)  with  A  =  1.) 
Suppose  that: 

(A6)  Pt[Uo  €  M  =  p{9,u)P,,[Uo  €  (fti] 

for  some  density  p{9,u)  that  is  continuously  differen¬ 
tiable  on  A.  Then,  the  likelihood  ratio  of  P$  with 
respect  to  P#,  on  P  =  ff{Un  :  0  <  n  <  ri)  is  simply 
given  by 

T,-l 

m=llp{9,Uj). 

j=0 

To  verify  (A2)  for  L{9)  in  this  setting  is  much  simpler 
than  the  verification  of  the  previous  section  for  Harris 
chains.  In  particular,  since  (2.5)  continues  to  hold, 
the  key  is  to  show  that 

E  sup  i{0y  <  oo  (2.6) 

for  some  c  >  0.  But 
E  sup  i(0y 

<TE^I^  sup  p(tf,C;o)’'P‘'’(ri  =  n]. 

Since  £'sup||_|,|<,  p(tf,C/o)^'  can  typically  be  made 
arbitrarily  close  to  1  for  e  small  enough,  it  is  evident 
that  (2.6)  holds  if  J?exp(Ar|)  <  oo  for  |A|  sufficiently 
small  and  positive.  However,  this  can  be  verified  eas¬ 
ily  by  using  suitable  Lyapunov  functions.  (In  fact, 
the  Lyapunov  fimction  r  of  the  previous  secticm  can 
be  suitably  specialized.)  By  verifying  (2.6),  this  per¬ 
mits  us  to  establish  that  u'(9o)  and  t'(0o)  can  be  es¬ 
timated  via  sample  means  of  YiL'(9o)  and  riL'(9o) 
respectively. 

However,  if  the  density  p(0,  x,  y)  of  Z  can  easily 
be  calculated,  the  derivative  L*(0o)  of  the  previous 
section  is  a  competing  estirrutor.  Note  that 

r,-J 

Eli(0)lC]^  Jl  p(0,Zj,Zi^O^L(0) 

where  <?  =  fl’(Zn  :  0  <  n  <  ri).  If  ((L(0o  +  h)  — 
i(0o))/h  :  0  <  h  <  c)  tM  Pt,  uniformly  integrable, 
then  (l(tfo  +  f»)  -  i(0o)/h  -  i'(0o)  in  L‘(P#,)  so 
(L'(P,.+k-L(tfo))/f»  -  £(i'(tfo)|inini‘(/’#.)-  It  fol¬ 
lows  that  the  difference  quotients  (L(0o+h)- L(0o))/h 
are  uniformly  integrable  as  demanded  by  (A2)  and 

L'(0o)  =  Z[t(0om 


By  the  principle  of  conditional  Monte  Carlo,  the  r.v. 
YiL'(0o)  has  smaller  variance  then  yiZ'(^o)-  So,  us¬ 
ing  L'(0o)  is  statisticaUy  desirable. 

However,  our  discussion  also  shows  that  for  sys¬ 
tems  satisfying  stochastic  recursions,  the  easiest  way 
to  establish  (A2)  for  L(0)  may  be  to  instead  estab¬ 
lish  (A2)  for  L(0).  Hence,  introducing  L(0)  can  be  a 
useful  theoretical  tool. 

3  STEADY-STATB  IPA 

We  now  turn  to  steady-state  derivative  estimation  us¬ 
ing  IPA,  focusing  on  discrete-time  problems.  Here, 
too,  regenerative  structure  plays  a  key  role  in  the  con¬ 
vergence  and  consistency  of  the  derivative  estimates. 

We  begin  with  some  background  on  IPA,  then 
give  conditions  for  the  derivative  estimates  to  be  re¬ 
generative,  and  finally  use  regenerative  structure  to 
prove  strong  consistency. 

3.1  IPA  Estimates 

Many  discrete-time  sequences  studied  through  simu¬ 
lation  satisfy  recursions  of  the  general  form 

=  n>0,  (1) 

where  (Un,n  >  0}  are  inputs  to  the  simulation  and 
{Wn,n  >  0}  are  the  outputs  of  interest.  We  allow 
the  Wn’s  to  be  d-vectors  and  the  Un*  to  be  /-vectors. 
With  this  generality,  (1)  is  by  no  means  restrictive; 
but  to  use  IPA,  we  wiU  need  to  put  further  conditions 
on  d- 

A  familiar  example  of  (1)  is  the  Lindley  equation 
for  the  waiting  times  in  a  single-server  queue: 

W^«+i  =  [Ws+5„-A„r,  n>0,  (2) 

where  Sn  is  the  n-th  service  time  and  A„  is  the  time 
between  the  n-th  and  (n  -f  l)-st  arrivals.  In  this  ex¬ 
ample,  d  =:  1,  /  =:  2,  and  Un  -  (An,5n);  d  is  defined 
by  (2). 

Suppose  now  that  each  C/n  is  a  (random)  function 
of  a  parameter  0  ranging  over  an  interval  [a,t];  then 
each  W„,  n  >  1,  depends  on  0,  and  we  may  also 
assume  that  Wq  depends  on  0.  We  want  to  compute 
derivatives  with  respect  to  this  parameter.  Let  Un,i 
and  Wn,i  denote  the  t-th  components  of  Un  and  Wn- 
Formally  differentiating  (1),  we  get 


t  s  where  the  partial  derivatives  of  the  t- 

th  component  of  i  are  with  respect  to  the  indicated 
components  of  its  arguments  and  are  ail  evaluated 
at  (Wn{0),Un{9)).  This  is  another  recursion,  map* 
ping  to  combined  with  (1), 

it  gives  a  mapping  from  (W,,  W")  to  (PV;+i,  W^^,) 
with  input  (Un,Ul^).  Equation  (3),  when  valid,  de¬ 
fines  an  IPA  algcffithm  for  computing  {W|^,  n  >  1}. 

Returning  to  the  Lindley  equation  (2),  we  find 
that  the  ^  implicitly  defined  there  is  not  a  differen¬ 
tiable  function:  differentiability  fails  where  +Sn— 
An  =  0.  For  the  queue,  this  event  corresponds  to 
one  busy  period  ending  at  exactly  the  same  time  the 
next  one  begins.  This  phenomenon  is  typical  of  many 
discrete-event  systems:  differentiability  may  fail  when 
two  events  occur  simultaneously,  and  these  are  often 
the  only  points  of  non-differentiability.  So,  justifica¬ 
tion  of  (3)  requires  some  care.  Fortunately,  this  is 
more  a  theoretical  than  a  practical  concern;  dif¬ 
ferentiability  is  often  assured  by  conditions  implying 
that  events  occur  singly  with  probability  one. 

We  now  give  a  set  of  conditions  from  Glasaer- 
man  (1992ab)  justifying  (3)  and  further  implying  that 
^[^4(0)1  =  These  conditions  will  also  be 

useful  in  our  analysis  of  steady-state  derivative  esti- 
maticm. 

Recall  that  a  function  f  :  S  S  C  \s 

Liptekiti  if  there  exists  a  constant  kj ,  called  a  mod¬ 
ulus,  such  that 

ll/(*)-/(y)ll  <*/ll*-y||.  *.y€S. 

Lipschits  functions  are  differentiable  almost  every¬ 
where.  A  Lipschits  functioo  of  a  scalar  is  absolutely 
continuous  and  is  therefore  the  indefinite  integral  of 
its  (almost-everywhere  defined)  derivative.  The  class 
of  Lipschits  functions  is  just  broad  enough  to  include 
min,  max,  and  similar  functions  arising  in  discrete- 
event  systems,  and  just  smooth  enou^  to  be  com¬ 
patible  with  IPA. 

We  call  a  random  function  X  =  {X(t),  0  €  (a,  (]} 
almost-surely  Lipschits  if  its  sample  paths  are  Lipe- 
chits  with  probability  one.  If  X  is  Lipschits,  let  Kx 
be  a  (random)  modulus  bt  X. 

We  now  proceed  with  the  conditions.  Our  first 
assumption  puts  minimal  smoothness  conditions  on 
the  inputs  and  on  the  initial  state: 

( Al)  Wo  and  {Un,  n  >  0}  are  aa.  Lipschits  functions 
on  [a, 6].  For  each  0  6  (a, 6],  Wq  and  {Un,n  >  0}  are 
aa.  differentiable  at  0,  taking  one-sided  derivatives  at 
the  endpoints. 


The  first  part  of  (Al)  restricts  dependence  on  9 
for  a  fixed  sample  path;  the  second  part  fixes  9  and 
varies  the  sample  path.  Both  types  of  conditions  are 
needed.  We  also  assume 

(A2)  ^  is  Lipschits. 

By  itself,  (A2)  implies  that  the  partial  derivatives 
of  4  exist  almost  everywhere.  But  this  is  not  quite 
enough  for  (3),  since  it  is  possible  for  {(Wn, f/n),  n  > 
0}  to  return  infinitely  often  to  the  null  set  of  non- 
differentiable  points  of  with  positive  probability. 
To  rule  this  out,  define 

=  {z  €  X  R*  :  ^  is  differentiable  at  z), 

and  require 

(A3)  P((W„(9),£r„(tf))  6  C^)  =  1,  for  aU  n  >  0,  for 
alltfe(a,k]. 

This  is  not  a  primitive  condition,  in  the  sense 
that  the  distributions  of  {(Wn(tf),CfB(9)),n  >  0)  are 
generaUy  unknown.  Nevertheless,  (A3)  is  often  easy 
to  verify  in  practice.  For  example,  in  the  Lindley  re¬ 
cursion  (2),  is  the  complement  in  A  x  A’  of  the 
set  {(»,s,a)  :  tt  +  s-a  =  0).  IL  say,  {5n,n  > 
0}  and  {Aa.h  >  0}  are  ii.d.  and  mutually  inde¬ 
pendent,  and  if  either  A«  or  5o  has  a  density,  then 
{(W;,5„,  A„),n  >  0)  never  leaves  C^,  aa. 

We  now  combine  these  conditions  to  validate  IPA 
estimates  for  (1): 

Lenuna  3.1.  Suppose  (A1)-(A3)  hold.  Then  each 
W-.«  R  >  0,  is  aa.  Lipschits  on  [a,k].  If  Wq  and  {I7„, 
n  >  0}  have  integrable  moduli  Kw,  uid  {Xu,  •  R  > 
0},  then  at  every  0  €  (a,k)  fot  which  Wn(y)  is  inte- 
grable,  E[W„(y)]'  exists  and  equals  E[W'(y)]. 

Pros/.  The  Lipschits  property  is  preserved  by  compo¬ 
sition,  so  under  (Al)  and  (A2),  Wn  is  aa.  Lipschits. 
Under  (Al)  and  (A3),  W»  is  also  differentiable,  aa., 
at  each  0.  Let  and  be  moduli  for  d  as  a  func¬ 
tion  of  its  first  and  second  arguments,  respectively,  for 
all  values  of  its  other  argument.  (For  example,  take 
=  k^,  t  =:  1,2,  the  modulus  guaranteed  by  (A2).) 
Simple  induction  shows  that 

liw;(y)ii  <  Kw,  2  k:,xKw,  +  J2k;yKu„  (4) 

and  Kw,  is  an  integrable  modulus  for  (each  compo¬ 
nent  of)  Wn.  Then,  by  dominated  convergence, 

limA-‘£(Wn>(#-»-k)-Wn.<(y)] 


exiiU  and  equal*  £[t^n.i(^)]>  >  =  1,  •  ■ .  ,d.  □ 

3.2  Regeneration 
Suppose,  now,  that  for  each  9, 

n-1 

n"‘  ^  -*  w{e),  aj.,  (5) 

i=0 

for  some  deterministic  function  u;(-).  Lemma  3.1  mo¬ 
tivates  c’u  examination  of  whether  similar  conditions 
imply 

(6) 

i=0 

There  are  two  considerations  in  (6)  —  whether  the 
limit  exists  and,  if  it  does,  whether  it  equals  v/{9). 
Regenerative  structure  is  particularly  useful  in  ad¬ 
dressing  the  first  question. 

While  convergence  in  (5)  is  also  often  baaed  on 
(possibly  implicit)  regenerative  properties,  it  turns 
out  that  a  somewhat  stronger  regenerative  structure 
is  usually  needed  for  the  convergence  in  (6).  Indeed,  it 
is  possible  to  have  all  the  sequences  W{9)  — 
{W|i(0),n  >  0},  0  €  [a,fr]  regenerate  simultaneously 
infinitely  often  and  yet  for  >  0}  to  fail  to 

be  regenerative  for  all  tf,  as  the  following  example  il¬ 
lustrates: 

Example  3.1  For  t  s  1,2,  let  s  >  1} 

be  the  points  of  two  independent,  unit-rate  Poissoo 
processes.  The  sequence  >  0}  is  the  set  of 

paints  of  a  rate-(l/0)  Poisson  process  if  9  >  0.  Let 
r(0)  be  the  superposition  of  these  points  and  Let 
be  1  or  2  depending  on  whether  the  n-th  point  of 
t{$)  is  from  or  Then 


rn(9), 

0, 


Xn 


2; 

1. 


Now  let  W(fi)  be  Um  sequence  of  q>adnp,  fVn(e)  = 
A  >  1.  Fn  each  t,  W{9)  is  an 
i.i.d.  sequence  (of  exponentially  distributed  random 
variables  with  mean  #/(#  -fr  !))•  Thos,  the  sequences 
{W{9),9  >  0}  trivially  tfgeatnU  smmltaneously  be¬ 
cause  each  regenerates  every  n.  However,  the  cor¬ 
responding  derivatives  are  given  by 

fWM  Xn.i  =  2,Z,  =  2; 

wi/As  _  J  Xn—i  —  l,^i«  —  1; 

1  Xn.,  =  l,Zn=2; 

Ur,.,(tf),  X,.,  =  2,Z,  =  l. 


The  first  two  cases  pose  ao  problem,  but  all  four  cases 
occur  infinitely  often,  and  the  last  two  show  that  there 
can  be  no  AT  for  which  the  distribution  of  W}f{6)  is 
independent  AT. 

When  {W4(tf),  n  >  0}  is  in  fact  regenerative,  it  is 
often  because  of  the  structure  present  in  the  following 
example: 

Example  3.2.  Consider,  again,  the  Lindley  recur¬ 
sion  (2).  Suppose  service  requirements  and  interar¬ 
rival  times  are  each  i.i.d.  and  mutually  independent 
of  each  other.  The  server  works  at  rate  1/9,  with 
0  <  <  £(i4i]/£(Si].  If  the  0-th  customer  finds  the 

queue  enq>ty,  then  Wo(0)  =  0  and  subsequent  waiting 
times  obey 

It  follows  that 

tr  +  ifW,(tf)-»-«5n>X„; 

.+iW-|0,  otherwise, 

with  Wo(tf)  =  0.  Thus,  {W4(tf),n  >  0}  returns  to 
aero  wheneirer  {W.(tf),n  >  0}  does;  the  state  (0,0)  is 
recurrent  for  the  Markov  chain  {(W’»(tf),  Wi(tf)),n  > 
0}. 

Hie  regeneration  in  Example  3.2  can  be  ex¬ 
plained  in  rough  terms  as  foUowa.The  process  { Wn(9), 
n  >  0}  returns  to  sero  infinitely  after*  because  many 
states  are  mapped  to  aero  by  d(-.5»,44«),  for  given 
(S;,,i4,).  In  particular,  if  ^(W;,(0),5n. An)  =0,  then 
the  same  is  true  throughout  a  neighborhood  of  Wn(9), 
a.a.  Similarly,  W'n.fi(tf)  remains  aero  under  a  suf¬ 
ficiently  small  change  in  9.  But  if  ^^>4.1  (9)  =  0 
throughout  a  neighborhood  of  9,  then  W44.i(tf)  =  0. 
Thus,  the  fact  that  the  waiting  times  comfit  from  dif¬ 
ferent  initial  states  forces  the  derivatives  to  equal  sero 
infinitely  often.  In  this  sense,  regeneration  at  aero  has 
special  significance  for  derivatives. 

To  formalise  these  ideas,  we  return  to  (3).  To 
write  this  recursion  more  compactly,  let  0^4  ud 
D^4  be,  respectively,  dxd  and  d  x  I  matrices  of  par¬ 
tial  derivatives  ol  4  with  respect  to  the  corresponding 
arguments.  Then  (3)  simplifies  to 

.»-[D,^(Wn(#).Cfn(®))K(«).  (7) 

This,  in  turn,  can  be  re-written  as 


(8) 


where  {Xi(^),n  >  0}  are  matrices  and  {y„(tf),n  >  0} 
are  vectors. 

This  representation  of  the  IPA  estimates  is  use¬ 
ful  in  establishing  regenerative  properties.  We  first 
give  a  result  in  the  setting  of  Harris  ergodic  Markov 
chains,  then  spedalise  to  classical  regeneration.  For 
background  on  Harris  chains,  see  Asmussen  (1987). 
Since  we  consider  just  one  value  of  9  at  a  time,  we 
suppress  the  argument. 

Theorem  3.2.  Suppose  {(W„,{7„,I7'),n  >  0}  is  a 
Harris  ergodic  Markov  chain  and  let  (Wq,  Uq,  Uq)  have 
the  invariant  distribution  of  this  chain.  If 

P{D^HWo,Uo)  =  0)>Q,  (9) 

then  [{Wn^Wl^fUn,Ul^),n  >  0}  is  a  Harris  ergodic 
Markov  chain. 

Procf.  That  {iW^,K.Un,Uk). 

n  >  0}  is  Markov  follows  from  (7)  and  the  hypothe¬ 
sis  that  {{Wn,Un,Un)<^  ^  0}  Markov.  Let  Xn  = 
Un)  and  let  By  Har¬ 
ris  ergodicity,  {iWn,Un,U!i), 

n  >  0}  couples  with  its  stationary  version  at  a  fi¬ 
nite  time  Nt,  aa.  Subsequently,  (A’n,yn)  coincides 
with  (Xn.yn);  i.e.. 

Condition  (9)  implies  that  s  0  for  some  finite 
Nj>  Ni,  a.s.  Then  =  IV,.  regardless  <rf  WJ; 

i.e.,  the  derivatives  couple  in  finite  time,  for  all  initial 
states. 

As  shown  in  Glaaserman  (1992b),  condition  (9) 
implies  that  n  >  0}  has  a  unique  stationary  dis¬ 
tribution,  giving  a  stationary  distribution  for 
{{Wn,W;^,Un,U'„),n  >  0).  But  any  Markov  chain 
that  admits  coupling  and  has  a  stationary  distriba- 
tion  is  Harris  ergodic.  o 

Similarly,  for  classical  regeneration  we  have 

Theorem  3.3.  Suppose,  now,  that  {(C/'«,C/'^),n  > 
0}  are  i.i.d.,  that  {WniO  >  0}  returns  to  the  origin 
infinitely  c^n,  aa,  and  that 

P(D.d(0,  Uo)  =  0,  l>.d(0, UoWi  =  0)  >  0.  (10) 

Then  {(W„ ,  W^), n  >  0}  is  regenerative  with  W'  =  0 
at  the  regeneration  times. 

Pnof.  If  Wn  =  0,  then  with  the  strictly  positive  prob¬ 
ability  in  (10),  =  0  and  (W^+i,  WJ^i)  becou'es 

independent  of  {(W*,  Wj),k  <  n}.  o 


With  either  the  Harris  ergodicity  of  Theorem  3.2 
or  the  classical  regenera:ion  in  Theorem  3.3,  we  have 
convergence  in  distribution  of  {W|^,n  >  0}  to  the 
stationary  distribution  Wg.  If  is  integrable,  then 

(11) 

<=o 

converges  almost  surely,  and  this  is  half  of  what  we 
need  for  (6). 

3.3  Consistency 

Once  we  have  aa.  convergence  of  time-averages  of 
^  0},  the  question  of  consistency  reduces 
to  one  of  interchanging  a  limit  and  a  derivative.  We 
will  give  two  seta  of  sufficient  conditions  for  this  in¬ 
terchange. 

For  (6)  to  hold,  we  need  conditions  on  the  depen¬ 
dence  cf  the  inputs  across  different  values  of  B.  The 
simplest  assumption  is 

(A4)  {Cr„,  n  >  0}  ate  i.i.d.  funciiont  on  [a,b]. 

Naturally,  (A4)  implies  that  {17' ,  n  >  0}  are  also 
i.i.d.  functions.  More  generaUy,  we  could  requite  that 
{{Wn,Un,Un),ri  >  0}  be  a  Harris  ergodic  function¬ 
valued  Markov  chain. 

Our  ^t  strong  consistency  result  is  based  on 
the  method  of  Hu  (1992).  Hu  shows  that  limit  and 
derivative  can  be  interchanged  under  convexity  con¬ 
ditions. 

Theorem  3.4.  Suppose  the  conditions  of  Theorem 
3.3  bold  with  (A4)  replacing  the  ii.d.  condition  given 
there.  Suppose  that  (Al)  bolds  with  “Lipachits'’  re¬ 
placed  by  “convex.”  Suppose  further  that  4  is  in¬ 
creasing  and  convex.  If  £[|iyo(d)Q  <  oo  for  all  d 
and  ^|iyo(i)|]  <  00,  then  (6)  holds  at  almost  every 
«e(o,k). 

Fme/.  The  composition  of  an  increasing  convex  func¬ 
tion  with  a  convex  function  is  convex;  thus,  every 
Wn(-)  is  aa.  convex.  A  convex  function  on  a  closed 
interval  is  Lipschits,  hence  absolutely  continuous,  so 
we  have 

iyn(«»)  =  lyn(tfl)  +  r*  S-S.. 

Jti 

for  all  n  and  Bi,Bj.  Now  take  time  averages  of  both 
sides  and  let  the  time  horixon  increase  to  infinity.  The 
result  is 

£tW(#,)]  =  £[W(#,)).».  lim  rn-‘f;W7(d)dd. 


By  convexity,  Wn{0)  <  Wl^{b),  a.8.  By  dominated 
convergence,  v^e  may  therefore  interchange  limit  and 
integral  to  get 


E[Wo{02)]  =  E\Wo{0i)]  +  r  E[Wi{0)]  d0, 

J*x 


for  ail  01,  With  u;(0)  =  this  proves  that, 

a.e.  on  [a,  6],  u;'(0)  exists  and  equals  £[Wg(0)].  □ 

The  function  max  is  convex,  so  Theorem  3.4  can 
be  applied  to  the  Lindley  recursion  and  its  general¬ 
izations. 

Our  next  result  drops  the  convexity  requirement 
by  putting  a  stronger  condition  on  the  Lipschitz  prop¬ 
erty.  As  in  (4),  let  be  a  modulus  for  ^  as  a 
function  of  its  second  argument  for  any  value  of  its 
first  argument.  In  the  setting  of  Theorem  3.3,  let 
{rk,b  >  0}  be  the  renewal  process  of  returns  to  the 
origin. 

Theorem  3.5.  Assume  the  conditions  of  Theorem 
3.3  and  (A4).  Suppose  (A1)-(A3)  hold  with  integrable 
moduli  in  (Al).  Suppose  Wo  and  Wi  are  integrable. 
If  is, a  <  1  and  supi  £(n(0)  —  <  <»i  then  (6) 

holds  at  almost  every  0  €  [o,  6]. 

Pros/.  Consider,  for  simplicity,  the  non-delayed  case 
ro  =  0.  If  Tj  <n  <  rysi,  then  it  follows  from  (4)  and 
the  fact  that  It;.  =  0  that 


•*r,+l 


Hence,  if  is  3  S  taking  expectations  we  get 
^OF;(^)IO  <  By  the  last 

hypothesis  in  the  theorem,  £[||I^(0)ID  is  therefore 
bounded  uniformly  in  n  and  0.  Arguing  much  as  in 
Theorem  3.4  (but  taking  expectations  first),  this  al¬ 
lows  us  to  interchange  limit  and  integral  to  get 

£:[Wo(0a)l--BlWo(#i)] 

=  Sm  I  n-^YE\W;{0)]d0 

=  rEmt)]d0. 

since  the  limit  of  (II)  is  also  the  limit  of  its  expec¬ 
tation.  This  implies  that  w(0)  s  £[Wb(0)]  is  differ¬ 
entiable  at  almost  every  0,  with  w'(0)  =  E(W'(0)}. 


3.4  Remarks  on  Continuous  Time 

Though  we  have  only  considered  consistency  of  IPA 
for  discrete-time  processes,  similar  techniques  are  use¬ 
ful  in  continuous  time.  We  briefly  outline  how. 

Let  X  =  {Xt,  t  >  0}  have  a  countable  state  space 
and  suppose  X  changes  state  at  times  {r«,n  >  0}. 
Suppose  X  depends  on  0.  Assuming,  say, 

t-^jj[X.(0))<U^m{0),  (12) 

the  question  is  whethei 

J*'  f(X,md*]-*m'{0),  (13) 

for  some  {<„,  n  >  0}  increasing  to  infinity. 

A  first  step  in  showing  (13)  is  arguing  that 

^  p  fiX.m  d.  =  x;  f(Y^{0))[rU,  -  r'], 

iaO 

where  is  the  state  just  after  the  n-tb  transition. 
Techniques  from  Section  3.1  ate  useful  here  because 
this  finite-hocison  derivative  estimator  will  typically 
be  unbiased  only  if  the  state-transition  times  are  Lip- 
schits  functions  of  0. 

For  many  discrete-event  systems,  it  is  possible  to 
eupplement  the  system  state  with  the  time  remaining 
for  scheduled  events  to  obtain  an  augmented  process 
{(y^,Cn),n  >  0}  that  is  Markov.  Regeneratioa  of 
this  process  is  useful  in  estabHshing  (12).  To  ana¬ 
lyse  IPA  estimators,  it  is  convenient  to  consider  a  fur¬ 
ther  augmented  process  {(y„,Cn,  An),n  >  0},  where 
A  records  infi)rmxtion  about  dcrissftscs  of  scheduled 
event  times.  One  way  for  {(y,,Cn), n  >  0}  to  re¬ 
generate  is  fix  y  to  visit  a  rtats  in  which  an  entirely 
new  set  of  events  is  scheduled;  often,  this  corresponds 
to  a  system  returning  to  an  enq>ty  state.  When  new 
events  are  scheduled,  new  derivatives  are  generated, 
so  {(yn,Cn,An),n  >  0}  also  regenerates,  and  this  is 
an  important  step  in  verifying  (13).  As  in  Section  3.2, 
we  aee  here  a  connection  between  a  strong  form  of  re¬ 
generation  for  the  original  process  and  regeneration 
at  zero  for  the  derivatives.  A  detailed  treatment  of 
the  continuous-time  settiitg  is  given  in  Glasserman  et 
al.  (1991). 
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